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Abstract: 
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The purpose of this study was to determine whether or not survey medium (electronic 
versus paper format) has a significant effect on the results achieved. To compare survey 
media, responses from elementary students to British Columbia’s Satisfaction Survey 
were analyzed. Although this study was not experimental in design, the data set served 
as a rich source from which to investigate the research question. The methods included 
reliability, item mean, response rate, response completeness, and factor analysis com- 
parisons across survey media. From the analyses, the differences between electronic and 
paper media in this study appear to be minor, and do not seem to have a significant effect 
on overall results. In conclusion, the medium does not seem to overly affect response 
patterns and does not pose any threats to the validity or reliability of survey results. 
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Introduction 

The British Columbia (BC) Ministry of Education introduced its 
Provincial Satisfaction Survey in the 2001/02 school year. It was developed 
“as part of ongoing efforts to make the education system more accountable, 
and to increase the existing information that districts and schools use to 
evaluate and improve their performance” (British Columbia Ministry of 
Education, 2004). The survey measures student, parent, and staff satis- 
faction with areas such as achievement, human and social development, 
and safety. The BC Ministry of Education has been utilizing electronic sur- 
veying for its annual survey of student, staff, and parent satisfaction since 
the 2002/2003 school year. The move toward electronic surveying has 
been gradual, as each school district must assess their ability to manage an 
electronic survey, as must each school within those districts. However, in 
2004, a sufficiently large number of schools in British Columbia adminis- 
tered the survey in its electronic format to allow for comparisons between 
media. Researchers at the BC Ministry of Education have turned to the 
question of whether or not survey medium (electronic versus paper format) 
has a significant effect on the results achieved. By examining the effects 
of survey medium on responses, evidence of the validity of electronic 
surveying in comparison to traditional methods can be accumulated. 
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Literature Review 

Survey research methodology has been greatly affected by the advent 
of new computer technologies over the last thirty years (Dillman, 2000). 
One such technology is, of course, the Internet. The development and 
expansion of the Internet has created new possibilities for researchers; for 
instance, it is a new forum for survey administration. Surveys can now be 
completed online or via email. Similarly, new survey populations can be 
accessed very simply, as the Internet is available worldwide. 

Using the Internet in surveying has proven to be cost-effective as well 
(Moss & Hendry, 2002; see also Solomon, 2001; Dillman, 2000). Printing 
and shipping costs do not exist as they do for mail-in surveys and Internet 
surveying is not as labour-intensive as in-person or telephone surveys. In 
fact, once the survey is designed and posted to the Internet (or prepared 
for email delivery), very little effort is required to deliver it. 

Electronic surveying does not appear to represent a significant burden 
to respondents; in fact, a study in which participants filled out both a 
printed survey and an online one “found little difference in time or effort 
between the two surveys” (Kaye & Johnson, 1999, p. 325). Lastly, using 
the Internet has implications for the return and analysis of data once the 
surveying is finished. In particular, using electronic surveys eliminates the 
need to scan individual surveys or “the often error prone and tedious task 
of data entry” (Solomon, 2001, Introduction). 

However, researchers in all fields are still wary about the use of Internet 
technologies in surveying. In particular, there are concerns that the 
medium itself may create a bias in responses that would not exist in paper- 
and-pencil or telephone surveys (Nesbary, 2000). The results of research 
involving electronic surveying have been mixed. For instance, when con- 
sidering the issue of whether survey medium affected response rates, some 
researchers report that electronic surveys garnered more responses, while 
others found that the opposite was true (Moss & Hendry, 2002). 

There is a similar concern that the type of survey method used will 
affect the validity of the data collected. Researchers report that survey 
data collected online appears to have less missing or nonsensical data 
than paper- and-pencil surveys do (Nesbary, 2000). This suggests that in 
some ways, electronic surveying represents an improvement over more 
traditional methods. However, respondents may approach the two media 
quite differently, and this can have an effect on the validity of data. Some 
researchers have found that electronic surveys garner more detailed and 
longer comments in open-ended questions, speculating that “respondents 
may be more likely to be self-disclosing or less likely to respond in a socially 
desirable way because of the sense of distance associated with responding 
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on the Internet” (Daley, McDermott, Brown, & Kittleson, 2003, p. 117). 
However, others have noted that respondents tend to read things that are 
online more quickly, suggesting that they take less time to consider ques- 
tions as they complete a survey (Crawford, Couper, & Lamias, 2001). 

As well, certain questions may be more sensitive to the effects of 
different modes of administration than others. In their study of college 
student engagement, Carini, Hayek, Kuh, Kennedy, and Ouimet (2003) 
found while little difference between electronic and paper media existed 
on most scales, there was one notable exception. This was with regard 
to the scale on computing and information technology, on which a sub- 
stantial difference was found between the two modes, with web survey 
participants showing more favorable responses (see also Webster & 
Compeau, 1996). 

Finally, there are practical concerns to consider. A traditional paper 
survey will look the same regardless of who is looking at it. Electronic sur- 
veys, however, may change in appearance depending on the respondent’s 
computer settings (Dillman, 2000). For example, questions that fit across 
the screen of one computer may disappear off the edge of another. Similarly, 
while a survey may have been designed so that the text and background 
colors contrast with each other (for instance, black text on a white screen), 
some computer settings may display these colors differently, making the 
survey more difficult to read (Kaye & Johnson, 1999). Differences in tech- 
nology are also important; users with older computers may not be able to 
properly display surveys that are web-based (Dillman, 2000). 

Thus, it appears that there is the potential for differences to arise on 
paper versus electronic survey methods. However, there are many factors 
that must be considered in trying to determine whether these differences 
actually influence the results. In order to explore the question of whether 
survey medium has an effect on survey response, this study examines 
responses to the BC Provincial Satisfaction Survey. 
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Research Question 

In this article, we are concerned with the issue of whether or not intro- 
ducing a different survey medium has any significant effects on results. 
Because paper and web-based surveys are occurring simultaneously in the 
BC education system, this provides a natural experiment of survey modes 
for study. The survey is a census survey of all students in grades 4 1 , 7, 10, 
and 12, their parents, and school staff, including over 400,000 potential 
respondents. However, in order to ensure that the administration process 
of the survey was equivalent, only those surveys administered to elemen- 
tary students (who received more detailed instructions, regardless of the 
survey medium) are included in this study. Thus, this study is in contrast 
with much of the research that has been conducted thus far: first of all, its 
target population is children, not adults. As well, the overall population 
for the Satisfaction Survey is identifiable, which is unusual, especially for 
electronic surveys. 

The question of focus for this study, therefore, is what differences exist 
in response tendencies between paper and electronic survey methods? 
The next section describes the methods by which we have approached this 
question. 
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Methods 

Equivalence of Forms 

In order to ensure that the two survey media were equivalent, the 
researchers first examined each instrument and compared the process of 
administering and analyzing the survey to eliminate the possibility that 
procedural differences could account for any differences found in the 
survey results. Differences and similarities are explained in this section so 
that the analysis can be interpreted in the proper context. 

Survey Design 

Paper Survey 

The Satisfaction Survey consists of between 1 9-27 questions depending 
on the target group. While several questions are of a yes/no nature, the 
majority of questions utilize a 5-point likert-style response scale. In order 
to minimize the amount of time required, the surveys were kept short 
in length and required fewer than 20 minutes to complete. Each school 
determined the best way to administer the surveys within the set time 
frame. Figure 1 is an excerpt from the grade 4 paper-based survey. 

Figure 1 : Layout of Paper Survey 
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Electronic Survey 

Electronic surveys were designed to appear similar to the paper survey. 
However, because of the change in medium, there are some features that 
are different. For instance, electronic surveys require the use of a unique 
logon number to access the survey. The survey questions are separated into 
different screens, displaying four to five questions at a time. Respondents 
are asked to select their answers using a radio button. Responses can be 
changed but only one response can be selected at a time for any individual 
question. Like the paper survey, questions can be left blank. Respondents 
use a “next” button to progress through the survey screens. An indicator 
of progress through the survey is also provided in the lower left corner 
(e.g., “viewing questions 1-5 of 21”). Dillman (2000) provides an exten- 
sive list of web-based survey design principles. For example, he suggests 
that each question should be presented in a conventional format similar to 
that normally used on paper self-administered questionnaires; logon num- 
bers should be provided for limiting access only to people in the sample; 
respondents should not be required to provide an answer to each ques- 
tion before being allowed to answer any subsequent ones; and the use of 
graphical symbols or words that convey a sense of where the respondent is 
in the completion process should be utilized. 

The BC Satisfaction Survey meets the majority of these principles 2 . 
Figure 2 is an excerpt from the grade 4 electronic-based survey. 

Figure 2: Layout of the Electronic Survey 


l Do yo*j like school? 


? Do you like what you ore learning at «<hool? 


3 Do you try to do yo»»- host nt vhool’ 


4 r>o adults in the school treat all students fairly'? 


Do you (oadior'j help you with you lichoolwortc when you need It? 


2004 Student S urvey - G rade 4 


M, A JJ. 


r r r r 


r \ r \ r r \ r 


r r r r r 




i-T-L-A 


Does Survey Medium Affect Responses? Surveying in British Columbia Schools 


Walt, Atwood, & Mann, 

10 


Survey Process 

Table 1 compares the similarities and differences in the process and 
appearance, by survey type. Our interest here is in how these differences 
in process and appearance may affect the results achieved. The table is 
broken out into stages of the survey process ( distribution , administration, 
appearance/performance, responses, and the resulting data file). 

Table 1 : Comparison of the Survey Process, Paper and Electronic Media 


Survey Process 

Paper 

Electronic 

Distribution 

• Printing/addressing, shipping & 
mailing errors may occur 

• Photocopying of forms is possible 

• Corruption of forms (e.g. trading 
between schools, photocopy size 
and reflectivity, wrinkling, soiling) 

• Missing forms in the mail 

• Logon numbers and the website 
address sent by email to school 
administrators 

• Logon numbers can be mixed up 
among grades or across schools 

• Extra logon numbers are sometimes 
needed 

Administration 

• Schools are expected to distribute 
paper surveys 

• A script is provided to the 
administrator 

• Paper surveys must be collected 
and returned 

• Schools expected to distribute Logon 
Numbers, provide computer access, 
and direct students to the website 

• Internet access is necessary 

• A script is provided to the 
administrator 

• Students must be brought to a 
computer lab to complete the survey 

Appearance/ 

Performance 

• Single sheet of paper 

• Respondents fill in "bubbles" with a 
pencil to indicate their response 

• Completion is slower than electronic 

• Multiple screens 

• Respondents select responses by 
clicking the mouse over a radio 
button 

• Completion is faster than paper 

■ An indicator of progress through 
survey is provided 

Responses 

• No "rules" for data capture (e.g. 
respondents can fill in two circles, or 
mark halfway between circles, etc.) 

• Corruption of forms (e.g. changing 
question wording or response scales) 
is possible 

• Notes on surveys (e.g. doodles, names, 
comments) are hard to capture or use 

• "Rules" for data capture are possible 
(e.g. no double responses, no 
comments, no doodling etc.) 

■ Corruption of forms not possible 

• Capture of comments can be 
accommodated for school use 

• Additional District and School 
questions can be added at end 

Data File 

• Scan to data file 

• Mis-scanning (due to form corruption) 
can occur 

• Hand entry required where form 
corruption prevents scanning 
(increases error) 

■ Direct to data file (no scan) 

■ Limited error introduced 
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Sample 

For the purposes of this research, we chose to limit our sample to the 
2003/ 04 school year. An examination of the survey process made it evident 
that the methods of administration of paper and electronic surveys were 
most closely matched for student surveys. In order to reduce the poten- 
tially misleading effects of survey administration 3 , we have therefore lim- 
ited our sample to younger student respondents. Students in grades 3/4 
and 7 are thus included in the sample 1 . We also chose to limit our sample 
to only those schools that conducted all their student surveys electroni- 
cally or all of them by paper rather than those schools who used a mix of 
media. Finally, we selected standard schools for our sample, as participa- 
tion by non-standard schools 4 was optional and participation rates were 
therefore often much lower for non-standard schools. 

In the end, we had a sample of 66,683 student responses from 1,216 
individual schools. Table 2 shows the number of schools and the number 
of respondents by grade and survey type. 

Table 2: Number of Schools and Respondents by Grade and Survey Medium 


Grade 

Medium 

Number of Schools 

Number of Respondents 

Grade 3/4 

Electronic 

666 

23,473 

Paper 

452 

15,568 

Total 

1,118 

39,041 

Grade 7 

Electronic 

416 

15,052 

Paper 

347 

12,590 

Total 

763 

27,642 


Note: Due to the fact that some schools enroll both grades 3/4 and 7, the sum of schools in this table (1,881) 
is greater than the total number of schools included in the sample (1,216). 
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Variables 

Our main goal was to examine whether survey type (paper vs. elec- 
tronic) would have an effect on survey responses. Additionally, however, 
there were other factors that we needed to consider in order to ensure that 
the effects we observed were, in fact, due to the survey medium. In partic- 
ular, we hypothesized that the effect of survey type might be mediated by 
the age of the respondent; thus, the child’s grade is important to consider. 
Our analysis, therefore, treats grade 3/4 students and grade 7 students as 
separate respondent groups. 

In order to determine whether or not survey medium was affecting 
response rates, we chose to focus on five indicators. First, we felt that 
examining survey reliability would be important. Secondly, we looked 
at whether there were significant differences in the item means between 
survey types. We looked at response rates across survey medium, and 
we considered whether the type of survey used had an effect on overall 
response completeness (i.e., the number of missing values in a given 
survey response and patterns of missing response). We also conducted 
a factor analysis of each survey medium to see if there were any major 
differences in the way that individual items loaded, as a possible indication 
of different response patterns for each survey medium. We felt that these 
indicators were comprehensive and would allow us to comment with some 
certainty on our research question. 

Procedures 

Sample Comparison 

Before we could compare the survey instruments, it was important 
to determine whether the students who filled out paper surveys were 
sufficiently similar to those who filled out electronic surveys. This was 
necessary so that we could eliminate respondent differences as a possible 
explanation for any differences we might find between the two media. To 
determine if the samples were similar, we examined the proportion of 
boys and girls, proportion of Aboriginal and Non-Aboriginal students, the 
region in which respondents filled out the survey, and the average class 
size for both paper and electronic surveys. These variables represented 
the only demographic data available to us, as the survey was administered 
anonymously. 
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Reliabilities 

Using SPSS version 10.0, an internal consistency reliability analysis 
(Cronbach’s alpha) was conducted to determine the interrelatedness of 
the survey questions. The internal consistency reliability is an index of 
the extent to which the questions “hold together” in the sense that if one 
responds in one way to say question 1, then one would tend to respond 
similarly to other questions on the survey. This index can vary from a 
minimum of 0.0 to a maximum of 1.0. An alpha approaching 1.0 suggests 
that the questions comprising a survey do have high internal consistency. 

Item Means 

Overall item means for each instrument were calculated, as were means 
for each individual question on the surveys. Using a t-test, means for each 
question were compared to determine whether there was a significant dif- 
ference between electronic and paper media. Such information assists an 
analysis of differences between the two media by providing an indication 
of whether respondents exposed to a particular medium responded to par- 
ticular questions differently than those using the other medium. 

Completing multiple tests on a single sample can affect the likelihood 
of finding a relationship that is due to more than chance. Abdi (2007) 
explains: 

The more tests we perform on a set of data, the more likely we are to 
reject the null hypothesis when it is true... this is a consequence of the 
logic of hypothesis testing: we reject the null hypothesis if we witness a 
rare event. But the larger the number of tests, the easier it is to find rare 
events and therefore the easier it is to make the mistake of thinking that 
there is an effect when there is none (p.l). 

In order to address this possibility, one strategy is to use the Bonferroni 
correction, which adjusts alpha according to the number of tests conducted 
on one sample. While not the only method of adjusting for multiple tests, 
it is a generally accepted method that is relatively simple to carry out. 
The Bonferroni correction was used for item mean calculations, using the 
number of questions on each survey instrument as the denominator. In 
the Results section below, the adjusted alpha is reported along with the 
results of the t-tests. 

Response Rates 

The response rates by grade for each group of schools (paper and elec- 
tronic) within our sample were calculated by taking the total number 
of survey responses received and dividing it by the number of surveys 
intended for that population. 
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Patterns of Missing Response 

As well, patterns of missing response were examined. First, the average 
number of missing cases was calculated for each survey medium, to deter- 
mine whether respondents to one medium were more likely to miss 
answering questions than another. Secondly, 2x2 contingency tables of 
missing vs. non-missing response were compared between survey medium 
for each question, allowing us to explore whether a definite pattern exists 
for either medium, and determine any differences in patterns of missing 
response between media. This provides an indication of the extent to 
which survey medium affects respondent error. 

Factor Analysis 

Finally, we conducted a comparison of factor patterns between the two 
survey media. Factor analysis is used to identify a small number of factors 
that can represent relationships among sets of many interrelated variables . 
It is based on correlations between responses to individual variables (i.e., 
survey questions) and is used to identify one or more linear combination 
of variables that collectively explain as much of the co-variation among a 
set of variables as possible. In this paper, factor analysis is used to compare 
differences in patterns of response between the two survey media (paper 
or electronic). The data were separated into groups by grade and survey 
medium, and separate factor analyses were run for each sub-set of data. 
For each grade group, the component matrix was then examined for how 
similarly each individual question loaded into each factor. The number of 
factors extracted and the total variance explained were also examined. 

Exploratory factor analysis was used in this analysis. While we 
acknowledge that traditionally, confirmatory factor analysis (CFA) is the 
more common method of comparing invariance between two groups, we 
did not feel that we had a strong enough theoretical explanation for mul- 
tiple factors (i.e. those that go beyond the concept of ‘satisfaction’) to war- 
rant using CFA. Instead, we used a Procrustean rotation that allows for 
the factor matrix of one group to a ‘best fit’ structure of the second group 
to compare the two structures analytically. This method involves calcu- 
lating cosines among factors; these can then be used to rotate one matrix 
to its best-fit position with a target matrix. The resulting cosine correla- 
tions indicate whether the factors are invariant across the two groups (see 
Thompson, 2004 for a more detailed explanation of the use of Procrustean 
rotation to measure invariance) . 
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Findings 

Sample Comparison 

The sample selected for this research shows almost equal proportions 
between the electronic and paper surveys for boys and girls. The propor- 
tions of Aboriginal and non-Aboriginal respondents are also very similar. 
There is some difference in the proportions of electronic surveys to paper 
surveys by region. When differences in school size were examined for elec- 
tronic and paper surveys, it was found that the distributions of size were 
very similar. Table 3 provides a comparison of sample characteristics by 
survey medium. 

Table 3: Reliability Measures for Achievement Scales 


Demographic 

Electronic 

Paper 

Gender 

Boy 

50.36% 

50.77% 

Girl 

48.45% 

48.89% 

Unspecified 

1 .1 8% 

0.34% 

Region 

Lower Mainland 3 

48.83% 

51.17% 

Other 

66.73% 

33.27% 

Aboriginal 

Status 

Aboriginal 

1 2.68% 

11.31% 

Non-Aboriginal 

85.62% 

87.02% 

Unspecified 

1 .70% 

1 .67% 


a "Lower Mainland" refers to the most densely populated urban area of British Columbia, located in the 
southwest of the province. It comprises 1 5 separate school districts that serve this metropolitan area. 


The average school size for schools that administered the survey 
electronically was 73 students, compared to 71 students for paper-based 
administration. 
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Reliabilities 

The grade 3/4 internal consistency reliability measure for the elec- 
tronic survey was .76 and .75 for the paper survey. The grade 7 internal 
consistency reliability measure for the electronic survey was .84 and .81 
for the paper survey. Table 4 provides the internal consistency reliability 
of each medium by grade. 

Table 4: Reliability Measure by Survey Medium for Grade 3/4 and Grade 7 

Students 


Survey Medium 

Number of Items 

Number of Cases 

Alpha 

Grade 3/4 Students 

Electronic 

17 

21,782 

.76 

Paper 

17 

14,054 

.75 

Grade 7 Students 

Electronic 

18 

24,256 

.84 

Paper 

18 

13,328 

.81 


Item Mean Comparisons 

The average mean response on a 1-5 likert-type scale for the paper 
surveys administered to grade 3/4s was 4.16, while the average mean for 
the electronic surveys administered to these grades was 4.10. The average 
mean for the paper surveys administered to grade 7s (3.84) was almost 
identical to the electronic surveys administered to this grade (3.83). 

In addition to overall means, item by item mean comparisons were 
also conducted. As mentioned in the Methods section, the significance of 
item mean comparisons was determined using the Bonferroni correction 
to alpha; thus, for questions on both the grade 3/4 and grade 7 surveys, 
alpha = 0.003. Using the adjusted alpha, there was a statistical difference 
between paper and electronic media for twelve questions that grade 3/4s 
answered, and for eight questions that grade 7s answered. However, many 
of these showed only minimal effect sizes. Cohen’s convention for effect 
sizes sets a small effect size at .20, a medium size at .40, and a large size 
at .80. According to these conventions, of the 14 questions that were 
statistically different for grade 3/4s, only one showed a medium effect; 
all others showed small effects. Similarly, only one of the 9 statistically 
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different questions for grade 7s showed a medium effect. (Table 5, below, 
shows item by item t - tests and effect sizes for grade 3/4s and Table 6, next 
page, shows item by item means and effect sizes for grade 7s). 

The t - test measure of significance can be influenced by sample size, 
with larger sample sizes leading to increased significance levels. Therefore, 
it may be that differences between the paper and electronic media are of no 
practical significance (Marshall, 2007). This would certainly be supported 
by our finding that only one question had an effect size larger than 0.3. 

Table 5: Item Mean Comparison and Effect Sizes for Grade 3/4s 


Question 

T-Value 

dF 

P 

Mean 

Difference 

Effect 

Size 

"Do you like school?" 

3.89 

32729.96 

<0.001 

0.040 

0.04 

"Do adults treat all students fairly?" 

6.54 

38757 

<0.001 

0.065 

0.07 

"At school, are you learning about how to stay 
healthy?" 

-6.05 

32661.67 

<0.001 

-0.076 

0.06 

"Are you getting better at reading?" 

7.67 

31380.42 

<0.001 

0.067 

0.09 

"Are you getting better at writing?" 

12.04 

31528.2 

<0.001 

0.112 

0.12 

"Are you getting better at math?" 

4.68 

31997.34 

<0.001 

0.044 

0.04 

"Are you getting better at using computers?" 

32.59 

26548.92 

<0.001 

0.385 

0.37 

"At school, do you get exercise (for example, 
physical activity or sports)?" 

-2.87 

38724 

<0.001 

-0.024 

0.04 

"At school, do you respect people who are 
different from you (for example, think, act, or 
look different)?" 

2.83 

32323.34 

<0.001 

0.022 

0.04 

"At school, do you participate in activities 
outside of class hours (for example, clubs, 
dance, sports teams, music)?" 

9.52 

31691.89 

<0.001 

0.135 

0.10 

"Do you feel safe at school?" 

9.48 

31669.11 

<0.001 

0.091 

0.10 

"Do teachers care about you?" 

7.38 

31071.82 

<0.001 

0.061 

0.08 
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Table 6: Item Mean Comparison and Effect Sizes for Grade 7s 


Question 

T-Value 

dF 

P 

Mean 

Difference 

Effect 

Size 

"Do you try to do your best at school?" 

3.359 

31133.38 

0.001 

0.028 

0.04 

"Do adults treat all students fairly?" 

-6.842 

31318.62 

<0.001 

-0.077 

0.07 

"Do your teachers help you with your school- 
work when you need it?" 

-1 1 .685 

32507.49 

<0.001 

-0.105 

0.12 

"Are you getting better at reading?" 

-3.511 

31714.54 

<0.001 

-0.038 

0.04 

"Are you getting better at using computers?" 

30.254 

27447.92 

<0.001 

0.425 

0.32 

"Do you feel safe at school?" 

-7.134 

31813.77 

<0.001 

-0.075 

0.08 

"At school, are you bullied, teased, or 
picked on?" 

7.935 

31880 

<0.001 

0.092 

0.08 

"Do teachers care about you?" 

-6.524 

39819 

<0.001 

-0.072 

0.07 


Response Rates 

As a measure of differences in response tendencies, response rates 
were calculated for each medium and grade. The results are presented in 
Table 7. The response rates were higher in both cases for the electronic 
survey; however, the response rates were very high (90% and above) and 
very similar across media. 

Table 7: Response Rates by Survey Medium and Grade 


Survey Medium 

Number 

Distributed 

Number Returned 

Response Rate 

Grade 3/4 Students 

Electronic 

25,573 

24,473 

92% 

Paper 

17,187 

15,568 

91% 

Grade 7 Students 

Electronic 

1 6,41 1 

15,052 

92% 

Paper 

13,955 

12,590 

90% 
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Patterns of Missing Response 5 

As a measure of respondent error, the average percentage of missing 
cases for each survey medium was calculated by grade (Table 8). For all 
grades and media, the average percentage of missing cases was less than 
1%. For grade 3/4s, the average percentage of missing cases in electronic 
surveys was 0.94%, compared to 0.89% for paper surveys. For grade 7s, 
the average percentage of missing cases was 0.61% and 0.75% for elec- 
tronic and paper surveys respectively. 

Table 8: Average Percentage of Missing Cases by Survey Medium 



Survey Medium 

Grade 

Electronic 

Paper 

Grade 3/4 

0.94% 

0.89% 

Grade 7 

0.61% 

0.75% 


Patterns of missing response were determined first by creating a new 
dummy variable with values “response” and “non-response”. Chi-square 
tests of significance were then run for the resulting contingency tables, 
comparing electronic and paper media for each question (grade 3/4 and 
grade 7 survey responses were run separately.) To determine the effect size 
of the difference for these questions, odds ratios were calculated. These 
predict the likelihood of missing a question on the electronic survey as 
compared to the paper survey. Table 9 (next page) summarizes the ques- 
tions for which significant differences were reported for grade 3/4, and 
shows the odds ratios associated with each. Table 10 (page 21) presents 
the same information for grade 7 responses. 
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Table 9: Missing Response Odds Ratios, Grade 3/4 Responses 


Question 

Chi 

dF 

P 

Odds 

Ratio 

"Do adults treat all students fairly?" 

24.997 

1 

<0.001 

0.55 

"Do your teachers help you with your schoolwork when 
you need it?" 

5.866 

1 

0.015 

1.39 

"At school, are you learning about how to stay healthy?" 

1 9.084 

1 

<0.001 

0.62 

"At school, do you respect people who are different from 
you (for example, think, act, or look differently)?" 

6.910 

1 

0.009 

1.34 

"Do you feel safe at school?" 

10.357 

1 

0.001 

1.40 

"At school, are you bullied, teased, or picked on?" 

16.367 

1 

<0.001 

0.71 

"Do teachers care about you?" 

53.526 

1 

<0.001 

1.92 


Seven questions on the grade 3/4 survey showed a significant differ- 
ence between paper and electronic media in terms of missing responses. 
However, the size of this difference, estimated through the calculation of 
odds ratios, is relatively unsubstantial. The largest difference is for the 
question, “do teachers care about you?”, where respondents were 1.92 
times more likely to miss the question on the electronic form than on the 
paper form. As well, questions that measure generally the same concept 
are not skipped in the same way. For instance, students responding to the 
electronic form were more likely to miss the question, “do you feel safe at 
school?”, whereas they were less likely to miss the question, “at school, 
are you bullied, teased, or picked on?” Both of these questions provide an 
indicator of students’ perception of safety, and the questions follow one 
another on both forms. 
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Table 1 0: Missing Response Odds Ratios, Grade 3/4 Responses 


Question 

Chi 

dF 

P 

Odds 

Ratio 

"Do you try your best at school?" 

20.999 

1 

<0.001 

0.58 

"Do adults treat all students fairly?" 

53.286 

1 

<0.001 

0.43 

"At school, are you learning about how to stay healthy?" 

27.939 

1 

<0.001 

0.53 

"Are you getting better at math?" 

7.212 

1 

<0.001 

0.72 

"Are you getting better at using computers?" 

11.562 

1 

0.001 

0.70 

"At school, do you get exercise (for example, physical 
activity or sports)?" 

7.278 

1 

<0.007 

1.54 

"At school, do you respect people who are different from 
you (for example, think, act, or look differently)?" 

8.788 

1 

0.003 

1.56 

"At school, are you bullied, teased, or picked on?" 

10.334 

1 

0.001 

0.72 

"Do you know how your school expects you to behave?" 

32.020 

1 

<0.001 

0.60 


Nine questions on the grade 7 survey were found to have significant dif- 
ferences in the number of missing responses recorded between electronic 
and paper media. As with the grade 3/4 survey, the magnitude of these 
differences was rather small, with the largest difference occurring on the 
question, “do adults treat all students fairly”, where students replying to 
the electronic form were 0.43 times as likely to miss the question as those 
responding to the paper form. A somewhat more definite pattern may be 
emerging in grade 7, where questions relating to learning and achievement 
(trying one’s best at school, learning to stay healthy, and getting better 
at math and computers) show a lower likelihood of being missed on the 
electronic form. 

Factor Analyses 

For the Grade 3/4 students, both the electronic and the paper survey 
data sets produce three factors with eigenvalues greater than one. The 
total variance explained was 39.54% for electronic responses and 39.09% 
for paper responses. Table 11 (next page) and Table 12 (page 22) provide 
full details of the factor analyses for Grade 3/4 students. 
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Table 1 1 : Grade 3/4 Factor Analysis, Electronic Survey Respondents 



Component 

Question 

1 

2 

3 

"Do you like school?" 

.50 

.28 

.28 

"Do you like what you are learning at school?" 

.47 

.31 

.31 

"Do you try to do your best at school?" 

.14 

.58 

.00 

"Do adults in the school treat all students fairly?" 

.70 

.09 

.06 

"Do your teachers help you with your schoolwork when you need it?" 

.56 

.12 

.21 

"At school, are you learning about how to stay healthy?" 

.32 

.02 

.59 

"Are you getting better at reading?" 

.11 

.66 

.09 

"Are you getting better at writing?" 

.13 

.67 

.12 

"Are you getting better at math?" 

.14 

.58 

.08 

"At school, are you getting better at using computers?" 

.08 

.44 

.23 

"At school, do you get exercise (for example, physical activity or 
sports)?" 

.23 

.16 

.57 

"At school, do you respect people who are different from you (for 
example, think, act, or look different)?" 

.30 

.42 

.05 

"At school, do you participate in activities outside of class hours (for 
example, clubs, dance, sports teams, music)?" 

-.01 

.20 

.54 

"Do you feel safe at school?" 

.67 

.18 

-.06 

"At school, are you bullied, teased, or picked on?" 

-.46 

-.06 

.46 

"Do you know how your school expects students to behave?" 

.35 

.29 

.18 

"Do your teachers care about you?" 

.69 

.15 

.11 


Note: Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser 
Normalization. Rotation converged in 4 iterations. 
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Table 1 2: Grade 3/4 Factor Analysis, Paper Survey Respondents 



Component 

Question 

1 

2 

3 

"Do you like school?" 

.48 

.24 

.39 

"Do you like what you are learning at school?" 

.47 

.23 

.43 

"Do you try to do your best at school?" 

.10 

.63 

.04 

"Do adults in the school treat all students fairly?" 

.67 

.13 

.10 

"Do your teachers help you with your schoolwork when you need it?" 

.52 

.12 

.23 

"At school, are you learning about how to stay healthy?" 

.30 

-.03 

.60 

"Are you getting better at reading?" 

.50 

.61 

.25 

"Are you getting better at writing?" 

.11 

.60 

.24 

"Are you getting better at math?" 

.13 

.45 

.27 

"At school, are you getting better at using computers?" 

.04 

.18 

.47 

"At school, do you get exercise (for example, physical activity or 
sports)?" 

.21 

.15 

.52 

"At school, do you respect people who are different from you (for 
example, think, act, or look different)?" 

.23 

.58 

-.05 

"At school, do you participate in activities outside of class hours (for 
example, clubs, dance, sports teams, music)?" 

-.09 

.18 

.52 

"Do you feel safe at school?" 

.68 

.17 

.00 

"At school, are you bullied, teased, or picked on?" 

-.49 

-.13 

.36 

"Do you know how your school expects students to behave?" 

.25 

.45 

.09 

"Do your teachers care about you?" 

.65 

.14 

.16 


Note: Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser 
Normalization. Rotation converged in 6 iterations. 


J-T-L-A 


Does Survey Medium Affect Responses? Surveying in British Columbia Schools 


Walt, Atwood, & Mann, 


24 


Generally speaking, the same pattern of factor loadings can be identi- 
fied for both electronic and paper survey respondents at the Grade 3/4 
level. Three distinct factors, measuring school environment (component 1 
in Table 11, page 22, and Table 12, previous page), achievement or learning 
(component 2), and extracurricular activities (component 3) are identifi- 
able regardless of medium, and with the exception of two questions, the 
loading patterns are comparable. The two exceptions are the question, “do 
you know how your school expects students to behave?” and “At school, 
are you getting better at using computers?” The former question loads in 
component 1 (school environment) for electronic survey respondents and 
component 2 (achievement or learning) for paper survey respondents. 
The computer question loads on component 2 for the electronic group and 
component 3 for the paper group. 

For the Grade 7 students, the electronic survey data produced three 
factors with eigenvalues greater than one, while the paper survey data 
produced four factors with eigenvalues greater than one. For comparison 
purposes, a fourth factor was included for the Grade 7 electronic survey 
data. The total variance explained was 45.69% for electronic responses 
and 48.16% for paper responses. Table 13 (next page) and Table 14 (page 
25) show the Grade 7 factor analysis results. 
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Table 1 3: Grade 7 Factor Analysis, Electronic Survey Respondents 



Component 

Question 

1 

2 

3 

4 

"Do you like school?" 

.49 

.28 

.36 

-.03 

"Do you like what you are learning at school?" 

.53 

.35 

.29 

-.10 

"Do you try to do your best at school?" 

.27 

.20 

.56 

-.08 

"Do adults in the school treat all students fairly?" 

.74 

.15 

.05 

.14 

"Do your teachers help you with your schoolwork when you need 
it?" 

.66 

.25 

.03 

.12 

"At school, are you learning about how to stay healthy?" 

.36 

.46 

.07 

-.02 

"Are you getting better at reading?" 

.21 

.77 

.13 

.02 

"Are you getting better at writing?" 

.21 

.73 

.21 

.06 

"Are you getting better at math?" 

.20 

.54 

.23 

.08 

"At school, are you getting better at using computers?" 

.14 

.62 

.07 

.03 

"At school, do you get exercise (for example, physical activity or 
sports)?" 

.20 

.29 

.44 

.15 

"At school, do you respect people who are different from you (for 
example, think, act, or look different)?" 

.36 

.00 

.55 

.06 

"At school, do you participate in activities outside of class hours 
(for example, clubs, dance, sports teams, music)?" 

-.10 

.16 

.72 

.09 

"Do you feel safe at school?" 

.52 

.18 

.19 

.51 

"At school, are you bullied, teased, or picked on?" 

-.06 

-.02 

-.04 

-.91 

"Do you know how your school expects students to behave?" 

.55 

.16 

.35 

.01 

"Do your teachers care about you?" 

.76 

.23 

.11 

.09 

"Are you aware of the school goals for improving student 
learning?" 

.48 

.31 

.31 

-.01 


Note: Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser 
Normalization. Rotation converged in 7 iterations. 
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Table 1 4: Grade 7 Factor Analysis, Paper Survey Respondents 



Component 

Question 

1 

2 

3 

4 

"Do you like school?" 

.47 

.45 

.11 

.07 

"Do you like what you are learning at school?" 

.52 

.43 

.15 

.00 

"Do you try to do your best at school?" 

.18 

.66 

-.05 

.02 

"Do adults in the school treat all students fairly?" 

.72 

.12 

.06 

.14 

"Do your teachers help you with your schoolwork when you need 
it?" 

.64 

.07 

.19 

.13 

"At school, are you learning about how to stay healthy?" 

.45 

.11 

.42 

-.08 

"Are you getting better at reading?" 

.15 

.64 

.33 

-.05 

"Are you getting better at writing?" 

.13 

.69 

.31 

.03 

"Are you getting better at math?" 

.11 

.58 

.22 

.11 

"At school, are you getting better at using computers?" 

.21 

.15 

.55 

-.15 

"At school, do you get exercise (for example, physical activity or 
sports)?" 

.25 

.06 

.64 

.09 

"At school, do you respect people who are different from you (for 
example, think, act, or look different)?" 

.31 

.41 

-.11 

.13 

"At school, do you participate in activities outside of class hours 
(for example, clubs, dance, sports teams, music)?" 

-.06 

.16 

.62 

.17 

"Do you feel safe at school?" 

.50 

.18 

.09 

.54 

"At school, are you bullied, teased, or picked on?" 

-.04 

-.04 

-.04 

-.88 

"Do you know how your school expects students to behave?" 

.50 

.31 

.04 

-.04 

"Do your teachers care about you?" 

.75 

.16 

.14 

.07 

"Are you aware of the school goals for improving student learn- 
ing?" 

.47 

.31 

.22 

-.12 


Note: Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser 
Normalization. Rotation converged in 6 iterations. 
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For Grade 7 students, four factors were identified: the same three com- 
ponents initially identified for Grade 3/4 students (school environment, 
achievement or learning, and extracurricular activities) and an additional 
factor having to do with safety. Again, patterns of response were similar 
across the survey media. However, four questions loaded differently for 
electronic respondents versus paper respondents. The question, “Do you 
try your best at school?” loaded as component 3 (extracurricular activi- 
ties) for electronic survey respondents, and as component 2 (achievement 
or learning) for paper survey respondents. “At school, are you learning 
to stay healthy?” loaded as component 2 for the electronic group and as 
component 1 for the paper group. Like the Grade 3/4 students, the ques- 
tion, “At school, are you getting better at using computers?” loaded as an 
achievement or learning question for electronic survey respondents, but 
as an extracurricular activity question for paper respondents. Finally, “At 
school, do you learn to respect people who are different from you?” loaded 
as component 3 for the electronic group and as component 2 for the paper 
group. 

To determine whether the factor structures of paper versus electronic 
responses were invariant, a Procrustean rotation was used to fit the struc- 
ture of electronic responses to the best-fit position with that of paper 
responses. If the models are invariant, then the factor cosines from the 
first group will be strongly correlated with those of the second group, indi- 
cating a similar underlying structure. This provides a statistical measure 
of invariance to complement the analytical evidence discussed above. 

For grade 3/4 students, there was little detectable variance in the 
rotated factor structures. Table 15 shows that the three factors that 
emerged from the electronic survey factor analysis are highly correlated to 
those that emerged from the paper survey analysis: there is a 0.997 corre- 
lation between the first factors for each group; a 0.983 correlation between 
the second factors, and a 0.986 correlation between the third factors. 

Table 1 5: Factor Cosine Correlations for Grade 3/4 Paper and 

Electronic Surveys 



Factor 1-B 

Factor 2-B 

Factor 3-B 

Factor 1-A 

0.997 

0.077 

0.030 

Factor 2-A 

- 0.081 

0.983 

0.165 

Factor 3-A 

- 0.017 

- 0.166 

0.986 
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Grade 7 responses also demonstrated a high level of invariance. Table 
16 shows that the four factors that emerged from the electronic survey 
factor analysis are highly correlated to those that emerged from the paper 
survey analysis. However, in this case, the second and third factors have 
been reversed. That is, while the first factors correlate with each other at 
0.981 and the fourth factors correlate with each other at 0.922, the second 
and third factors do not correlate with each other. Instead, the second 
factor of the electronic survey analysis correlates with the third factor of 
the paper survey analysis at 0.935, while the third factor of the electronic 
survey analysis correlates with the second factor of the paper survey anal- 
ysis at 0.938. 


Factor Cosine Correlations for Grade 7 Paper and 
Electronic Surveys 



Factor 1-B 

Factor 2-B 

Factor 3-B 

Factor 4-B 

Factor 1 -A 

0.981 

0.105 

- 0.162 

0.018 

Factor 2-A 

0.135 

0.221 

0.935 

- 0.244 

Factor 3-A 

- 0.126 

0.938 

- 0.125 

0.299 

Factor 4-A 

0.057 

- 0.248 

0.291 

0.922 


This suggests that some variance in factor ordering has occurred. As 
well, while all the factors from the electronic survey analysis correlated to 
those in the paper survey analysis, the correlations are slightly less strong 
as were found for the grade 3/4 responses. Further research into the causes 
of the reversal of factor cosine correlations could reveal subtle variances in 
the factor structures of each medium. 
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Discussion 

The purpose of this study was to examine whether the medium of the 
Provincial Satisfaction Survey had any significant effect on its results - in 
other words, to compare the two media and look for differences that might 
suggest one medium was less reliable than the other. 

Generally speaking, our results have shown that this is not the case. 
The average number of missing cases in either medium is negligible. Both 
media had high reliability, and the difference between media was minor. 
Electronic surveying was slightly more reliable for both grade 3/4’s and 
grade 7’s, despite the fact that there were a slightly higher number of 
average missing cases in surveys completed electronically. The higher reli- 
ability is likely due to the reduction of error in other aspects of the survey 
process - for instance, there are no opportunities for data entry or scan- 
ning errors with electronic surveys. Practically speaking, it appears that 
electronic surveys hold a slight advantage over paper in that the simplicity 
of administering them contributes to a greater overall reliability. 

Item by item mean comparisons bear out this trend: although a 
number of questions in each grade were statistically different depending 
on medium, the size of these differences was negligible. The only exception 
to this was the question, “at school, are you getting better at using com- 
puters?” - a medium effect size was found for the mean difference on this 
question, with students completing an electronic survey reporting higher 
levels of agreement than those filling out a paper survey. This is consis- 
tent with Carini, Hayek, Kuh, Kennedy and Ouimet’s (2003) finding that 
answering questions about computer technology while using that same 
technology will result in higher satisfaction. Even so, it should be noted 
that students’ responses were quite positive, regardless of the survey 
medium. 

Over-inflation of statistical significance may be an issue due to 
the large sample size. This means that some of the questions for which 
statistical differences between media were found are not substantively 
different. This reinforces the trend of our findings, where we have seen little 
evidence to suggest that survey medium meaningfully affects response 
patterns. 

Both survey media have high response rates, and there is very little 
difference between the two. Electronic surveys tended to have higher 
response rates (by 1 or 2%) in both grades; again, this likely is related to 
the method of administering the survey: while a student may write all over 
his or her paper survey, rendering it impossible to scan and thus impos- 
sible to analyze, electronic surveying limits this kind of activity. The ‘rules’ 
for responding are programmed into the design of the survey, such that a 
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question may be answered or skipped, but no additional information (such 
as comments or drawings) that might render a response invalid can be 
included. 

When patterns of missing response are analyzed, while seven ques- 
tions on the grade 3/4 survey and nine questions on the grade 7 survey 
have significant differences in non-response between electronic and 
paper forms, these differences are small in magnitude. There appears to 
be little evidence that a distinct pattern of non-response related to a par- 
ticular subject or the order of questions on the survey instrument exists, 
although a pattern of fewer missing responses on the electronic form for 
questions relating to learning and achievement may be emerging in grade 
7 responses. Given the small number of missing cases overall, however, 
it is impossible to confirm whether a pattern exists, or whether the dis- 
tribution of missing cases is random, especially given the likelihood that 
chi-square statistics will be significant in large sample sizes. 

Generally, the factor analyses indicated little substantive difference 
between the patterns of response on electronic versus paper surveys. 
Only two questions for Grade 3/4 students and four questions for Grade 
7 students loaded on different factors for electronic surveys than they 
did for paper surveys. Most notable of these is, again, the question about 
computer satisfaction. For those respondents who filled the survey out 
on a computer, the satisfaction question regarding computers loaded on 
the ‘achievement or learning’ factor. Paper survey responses resulted in 
this question loading on the ‘extracurricular activities’ factor. This is not 
surprising. Students who were filling out the questionnaire electronically 
were engaged in using computer technology for school-related work in 
that the survey was completed using the school computer lab, and admin- 
istered by school staff. Thus, it would make sense that they viewed com- 
puters as part of their education. Those who filled out the paper surveys 
may not have been thinking of the use of computers as an educational tool 
when answering the questions, especially given the number of recreational 
uses for children that computers could represent. As with item means, this 
measure of comparison suggests that responses to questions related to 
computer technology do vary depending on survey medium. 

When the two media were rotated to best-fit position with each other, 
very little invariance was revealed. For the grade 3/4 surveys, the factor 
cosines for each emerging factor were highly correlated, suggesting an 
invariant underlying factor structure. For the grade 7 surveys, the correla- 
tion between factors remained high, but not as high as those of the grade 
3/4 responses, and two of the factors (factors 2 and 3) were reversed, such 
that factor 2 of the electronic survey analysis correlated with factor 3 of 
the paper survey analysis and vice versa. This suggests that there may be 
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some difference in the ordering of factors between the paper and electronic 
surveys; further research would need to be conducted in order to account 
for that discrepancy. 

In short, the differences between electronic and paper media for this 
study appear to be minor, and do not seem to have a significant effect on 
overall results. This bodes well for the use of electronic surveying, since elec- 
tronic methods of administration do have practical advantages over tradi- 
tional paper surveys: they are easier to administer, are more cost-effective, 
allow for shorter turnaround times in data processing, and require less 
coordination between different groups (such as the departments of edu- 
cation, school districts, schools, mail services, and couriers) than paper 
surveys do. Since the administration of the survey, for younger students 
at least, is conducted nearly the exact same way with each medium, biases 
resulting from the survey medium appear to be minimal. 

Our findings are consistent with recent research into the use of elec- 
tronic forms for testing and assessment in schools; for example, Poggio, 
Glasnapp, Yang and Poggio (2005) found very little difference in perfor- 
mance between computer based and paper and pencil test media. Similarly, 
electronic surveying stands up to the test: it is just as valid and reliable in 
these circumstances as traditional paper surveys. 


Limitations 

Our conclusions must be tempered by an understanding of the limita- 
tions of this study. One important limitation is that this was not a true 
experimental design. In a true experiment, a representative sample of 
students would have been selected and randomly assigned to either the 
paper or the electronic survey medium. This study, however, capitalizes on 
data already being collected for a different purpose. The British Columbia 
Provincial Satisfaction Survey aims to be a census of students in particular 
grades; it is designed to be of use primarily at the school level, rather than 
at an aggregated provincial level (although provincial results are certainly 
reported). Electronic surveying was implemented not to determine the 
validity of the method, but as an alternative method of survey administra- 
tion that might potentially increase accessibility and decrease costs. 

A further consideration had to do with the way in which electronic 
surveying was implemented. Individual districts chose whether or not elec- 
tronic surveying would be used; after this decision was made, individual 
schools still had the option to opt out and use paper surveys if they felt it 
was the better administrative choice. It is conceivable that differences that 
seem to be due to survey type could actually be a result of the decisions of 
the district or school in which a respondent is located. 
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Indeed, of the sample variables collected, only region was substantially 
different, with more regions outside of the Lower Mainland opting to use 
electronic surveying than those inside the Lower Mainland. This suggests 
that there may be some differences between school districts, particularly 
around the availability and accessibility of technology required to imple- 
ment electronic surveying. However, this ‘self-selection’ is happening at 
the school level, not the respondent level, which may minimize the effect 
at the individual level with which our analysis is concerned. 

A final limitation of this study is that we did not conduct a confirma- 
tory factor analysis in order to undertake a comprehensive evaluation of 
structural invariance. We feel that since the factor analyses were meant to 
supplement our other findings and were therefore not the central measure 
of comparison in this study, the Procrustean rotation on our exploratory 
factor analyses provides enough evidence to support our conclusions. 
However, this is not to suggest that a more thorough investigation of mea- 
surement invariance would not reveal subtle distinctions between the two 
survey media. Indeed, we believe that these distinctions, if they existed, 
could reveal important differences between paper and pencil surveys and 
electronic versions, especially when considering respondent groups that 
are not as familiar with surveying in general, such as children. A complete 
confirmatory factor analysis examination of measurement invariance 
would be, therefore, an excellent way to take this research further in the 
future. 

Conclusion 

In conclusion, it appears that electronic surveying and paper sur- 
veying produce very similar results. This suggests that the use of the elec- 
tronic medium to administer a large-scale survey like the BC Provincial 
Satisfaction Survey does not pose any threats to the validity or reliability of 
survey results. In fact, the use of electronic questionnaires may be advan- 
tageous, especially in surveying children; practical limits on responses pre- 
vent children from rendering their surveys unusable by drawing or writing 
in the response space and spoiling the form. Using computers is fun for 
children, making the survey a pleasant experience, and with the exception 
of questions about computers, the use of this medium does not seem to 
change or bias response patterns. 
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Endnotes 

1. Schools with no Grade 4 students administered the survey to Grade 3 students, 
their parents, and school staff. 

2. Dillman’s (2000) principles were developed with adult respondents in mind. Some 
of these principles were not applicable to surveying children, or to census surveys 
administered in school settings. 

3. Parent surveys, for example, were sometimes brought home by students to be 
filled out by parents and then returned. Sometimes they were mailed. Sometimes a 
password to access an electronic copy was sent home and sometimes parents came 
in to the school to fill out electronic surveys, and so forth. These different methods 
of administration were adopted by districts in order to maximize responses, and 
were determined purely by practical concerns. 

4. Non-Standard School: includes the following public school facility types: 
Continuing Education, Distance Education School - paper based program, 
Alternate Programs, Youth Custody, Long Term Provincial Resource Program, 
Electronic Program - in a district, and Electronic Program - Distance Education. 

5. Cases were coded as missing if: no response value was entered (electronic and 
paper surveys), multiple responses were entered (paper surveys only), or invalid 
responses were entered (paper surveys only). Invalid responses included such 
things as striking a line through the response options, marking a dot between 
two response options, and so forth. 
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